Deposit detection device and deposit detection method

ABSTRACT

A deposit detection device according to an embodiment includes a calculation module, an extraction module, and a decision module. The calculation module calculates a brightness average for each of small regions into which a predetermined region of an image captured by an imaging device is divided. The extraction module extracts, as a high-brightness region, the small region in which the brightness average calculated by the calculation module is equal to or larger than a predetermined value. The decision module decides on a brightness threshold value for detecting a deposit region corresponding to a deposit adhering to the imaging device, based on the area of the high-brightness region extracted by the extraction module.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2019-172215, filed on Sep. 20,2019, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are directed to a deposit detectiondevice and a deposit detection method.

BACKGROUND

Conventionally, a deposit detection device is known which detects aregion corresponding to a deposit adhering to a lens of an imagingdevice (hereinafter, deposit region), based on a brightness value of animage captured by the imaging device (for example, refer to JapaneseLaid-open Patent Publication No. 2006-157740).

When a deposit adheres to a large part of a lens, gain control toelevate the brightness of the entire image is sometimes performed in theimaging device. Since this gain control also makes the brightness of adeposit region higher, conventionally, a threshold value for detecting adeposit region is changed according to the brightness of the entireimage.

Unfortunately, conventionally, when gain control is performed in a statein which a deposit does not adhere, a region with low brightness may beerroneously detected as a deposit region.

SUMMARY

A deposit detection device according to an embodiment includes acalculation module, an extraction module, and a decision module. Thecalculation module calculates a brightness average for each of smallregions into which a predetermined region of an image captured by animaging device is divided. The extraction module extracts, as ahigh-brightness region, the small region in which the brightness averagecalculated by the calculation module is equal to or larger than apredetermined value. The decision module decides on a brightnessthreshold value for detecting a deposit region corresponding to adeposit adhering to the imaging device, based on the area of thehigh-brightness region extracted by the extraction module.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a diagram illustrating an overview of a deposit detectionmethod according to an embodiment;

FIG. 1B is a diagram illustrating an overview of the deposit detectionmethod according to the embodiment;

FIG. 2 is a block diagram illustrating a configuration of a depositdetection device according to the embodiment;

FIG. 3 is a diagram illustrating a process in a calculation module;

FIG. 4 is a diagram illustrating an example of threshold valueinformation;

FIG. 5 is a flowchart illustrating a procedure of the whole processperformed by the deposit detection device according to the embodiment;and

FIG. 6 is a flowchart illustrating a procedure of a detection processperformed by the deposit detection device according to the embodiment.

DESCRIPTION OF EMBODIMENTS

Embodiments of a deposit detection device and a deposit detection methoddisclosed by the subject application will be described in detail belowwith reference to the accompanying drawings. It should be noted that thepresent invention is not limited by the embodiments illustrated below.

First, referring to FIG. 1A and FIG. 1B, an overview of the depositdetection method according to an embodiment will be described. FIG. 1Aand FIG. 1B are diagrams illustrating an overview of the depositdetection method according to the embodiment. The upper section of FIG.1B illustrates an image I (hereinafter, captured image I) captured, forexample, in a state in which a light-blocking deposit such as dirtadheres to a lens of a camera (an example of imaging device) mounted ona vehicle. When such a light-blocking deposit adheres, a deposit regionin the captured image I is in a blocked-up shadow state.

As illustrated in FIG. 1A, a deposit detection device 1 according to theembodiment acquires a captured image I captured by a camera 10 mountedon a vehicle, for example, and detects a light-blocking deposit adheringto a lens of the camera 10 based on the captured image I. Thelight-blocking deposit is, for example, a region of dirt, insects, andthe like, in other words, a region in which a deposit region is ablocked-up shadow.

Here, the camera 10 may perform automatic gain control to elevate (makehigher) the brightness of the entire image, for example, when thebrightness of the entire image of the captured image I is low. In thiscase, as illustrated in FIG. 1A, in the captured image I subjected togain control, the brightness of the deposit region becomes higher andthe brightness of the region other than the deposit also becomes higher.

Conventionally, a threshold value for detecting a deposit region ischanged according to the brightness of the entire image, using thecharacteristic of gain control making the brightness of a deposit regionhigher. However, conventionally, when gain control to elevate thebrightness is performed in a state in which a deposit does not adhere,the threshold value of brightness is changed to be higher and thereforea region with low brightness or the like may be erroneously detected asa deposit region.

Then, in the deposit detection method according to the embodiment, athreshold value of brightness (hereinafter, brightness threshold value)is decided on based on the area of a high-brightness region with highbrightness in the captured image I after gain control. Referring now toFIG. 1B, an overview of the deposit detection method according to theembodiment will be described.

As illustrated in FIG. 1B, in the deposit detection method according tothe embodiment, first of all, an average value of brightness (brightnessaverage) is calculated for each of small regions 100 into which apredetermined region ROI of the captured image I is divided (step S1).The method of setting the predetermined region ROI and the small regions100 will be described later with reference to FIG. 3.

Subsequently, in the deposit detection method according to theembodiment, the small region 100 in which the calculated brightnessaverage is equal to or larger than a predetermined value is extracted asa high-brightness region 200 (step S2). The high-brightness region is,for example, a blown-out highlight region.

Subsequently, in the deposit detection method according to theembodiment, a brightness threshold value for detecting a deposit region500 is decided on based on the area of the extracted high-brightnessregion 200 (step S3). The area of the high-brightness region 200corresponds to, for example, the number of high-brightness regions 200.

In the deposit detection method according to the embodiment, the depositregion 500 is then detected using the decided brightness threshold value(step S4). Specifically, in the deposit detection method according tothe embodiment, when the small region 100 satisfies all of the followingconditions (1) to (3), the small region 100 is detected as the depositregion 500. When the small region 100 does not satisfy at least one ofthe conditions (1) to (3), the small region 100 is detected as anon-deposit region in which a deposit does not adhere.

(1) The difference in brightness average from the captured image I inthe past is equal to or smaller than a predetermined value.

(2) The difference in standard deviation of brightness from the capturedimage I in the past is equal to or smaller than a predetermined value.

(3) The brightness average of the small region 100 in the captured imageI at present is equal to or smaller than the brightness threshold value.

In this way, in the deposit detection method according to theembodiment, when the camera 10 performs gain control to elevate thebrightness of the entire image, a brightness threshold value is decidedon by using the characteristic of producing the high-brightness region200 having the area according to the degree of gain control.

In the deposit detection method according to the embodiment, therefore,since the degree of gain control can be grasped from the area of thehigh-brightness region 200, a brightness threshold value can be decidedon with high accuracy. That is, the deposit detection method accordingto the embodiment can detect a deposit with high accuracy.

In the deposit detection method according to the embodiment, abrightness threshold value is decided on based on the area of thehigh-brightness region 200 and the brightness of the entire image, whichwill be described in detail later with reference to FIG. 4.

Referring now to FIG. 2, a configuration of the deposit detection device1 according to an embodiment will be described. FIG. 2 is a blockdiagram illustrating the configuration of the deposit detection device 1according to an embodiment. As illustrated in FIG. 2, the depositdetection device 1 according to an embodiment is connected with thecamera 10, a vehicle speed sensor 11, and various equipment 50. Althoughthe deposit detection device 1 illustrated in FIG. 2 is a separatecomponent from the camera 10 and the various equipment 50, the depositdetection device 1 may be integrated with at least one of the camera 10and the various equipment 50.

The camera 10 is, for example, an on-vehicle camera including a lenssuch as a fish-eye lens and an imager such as a charge-coupled device(CCD) or a complementary metal oxide semiconductor (CMOS). The cameras10 are provided, for example, at positions where images at the frontside, the back side, the left side and the right side of the vehicle canbe captured, and output the captured images I to the deposit detectiondevice 1. The camera 10 performs gain control on the captured image I,if necessary, before outputting the captured image I to the depositdetection device 1.

The vehicle speed sensor 11 is a sensor that detects the speed of thevehicle. The vehicle speed sensor 11 outputs information on the detectedvehicle speed to the deposit detection device 1.

The various equipment 50 acquires the detection result from the depositdetection device 1 to perform a variety of control on the vehicle. Thevarious equipment 50 includes, for example, a display device indicatingthat a deposit adheres to the lens of the camera 10 and notifies theuser of an instruction to wipe off the deposit, a removal device thatejects fluid, gas, or the like toward the lens to remove the deposit,and a vehicle control device for controlling autonomous driving, forexample.

As illustrated in FIG. 2, the deposit detection device 1 according tothe embodiment includes a control unit 2, a volatile storage unit 3, anda nonvolatile storage unit 4. The control unit 2 includes apreprocessing module 21, a calculation module 22, an extraction module23, a decision module 24, a detection module 25, and a flag outputmodule 26. The storage unit 3 stores therein threshold value information31.

Here, the deposit detection device 1 includes, for example, a computerhaving a central processing unit (CPU), a read-only memory (ROM), arandom-access memory (RAM), a flash memory, and an input-output port,and a variety of circuits.

The CPU of the computer reads and executes a computer program stored inthe ROM, for example, to function as the preprocessing module 21, thecalculation module 22, the extraction module 23 the decision module 24,the detection module 25, and the flag output module 26 of the controlunit 2.

At least one or all of the preprocessing module 21, the calculationmodule 22, the extraction module 23 the decision module 24, thedetection module 25, and the flag output module 26 of the control unit 2may be configured by hardware such as an application specific integratedcircuit (ASIC) and a field-programmable gate array (FPGA).

The storage unit 3 corresponds to, for example, a RAM, a ROM, or a flashmemory. The RAM, the ROM, or the flash memory can store therein thethreshold value information 31, information on a variety of computerprograms, and the like. The deposit detection device 1 may acquire thecomputer programs and a variety of information described above throughanother computer connected via a wired or wireless network or a portablerecording medium.

The threshold value information 31 stored in the storage unit 3 isinformation including threshold values for detecting the deposit region500 by the detection module 25 described later, and a threshold value tobe used for a detection process by the detection module 25 is decided onfrom among these threshold values by the decision module 24 describedlater. The detail of the threshold value information 31 will bedescribed later with reference to FIG. 4.

The preprocessing module 21 performs predetermined preprocessing on thecaptured image I captured by the camera 10. The captured image Iacquired by the preprocessing module 21 is the image after automaticgain control is performed.

As a specific example of preprocessing, the preprocessing module 21performs a pixel thinning process on the acquired captured image I andgenerates an image having a size smaller than the acquired image. Thepreprocessing module 21 also generates an integrated image of the sumand the sum of squares of pixel values in the pixels, based on the imagesubjected to the thinning process. As used herein, the pixel value isinformation corresponding to brightness or an edge of a pixel.

In this way, the deposit detection device 1 can accelerate calculationin the processes in the subsequent stages by performing the thinningprocess on the acquired image and generating the integrated image andcan reduce the process time for detecting a deposit.

The preprocessing module 21 may perform a smoothing process for eachpixel, using a smoothing filter such as an averaging filter. Thepreprocessing module 21 does not necessarily perform the thinningprocess and may generate an integrated image of the captured image Ihaving the same size as that of the acquired image.

The preprocessing module 21 outputs the captured image I that is anintegrated image to the calculation module 22.

The calculation module 22 calculates brightness information indicating afeature amount of brightness for each of the small regions 100 intowhich a predetermined region ROI of the captured image I is divided.Referring now to FIG. 3, a process in the calculation module 22 will bedescribed. FIG. 3 is a diagram illustrating the process in thecalculation module 22.

As illustrated in FIG. 3, the calculation module 22 first sets thepredetermined region ROI and the small regions 100 in the captured imageI. The predetermined region ROI is a rectangular region preset accordingto the characteristics of the camera 10 and is a region, for example,excluding a vehicle body region and a housing region of the camera 10.The small regions 100 are rectangular regions formed by dividing thepredetermined region ROI in the horizontal direction and the verticaldirection. For example, each small region 100 is a region including40×40 pixels, but the number of pixels included in the small region 100can be set as desired.

Subsequently, the calculation module 22 calculates brightnessinformation indicating a feature amount of brightness for each smallregion 100. Specifically, the calculation module 22 calculates anaverage value of brightness (brightness average) and a standarddeviation of brightness (brightness standard deviation) as a featureamount, for each small region 100. The calculation module 22 alsocalculates a feature amount of brightness (an average value ofbrightness and a standard deviation of brightness) in the entirepredetermined region ROI.

Subsequently, the calculation module 22 calculates a variation infeature amount of brightness in the captured images I from the past tothe present. Specifically, the calculation module 22 calculates, as avariation, a first difference that is a difference in average value ofbrightness in the small region 100 at the same position in the past andat present in the captured images I. That is, the calculation module 22calculates, as a variation, the first difference between the averagevalue of brightness in the past and the average value of brightness atpresent for the corresponding small region 100.

The calculation module 22 also calculates a second difference that is adifference in standard deviation of brightness in the small region 100at the same position in the past and at present of the captured imagesI. That is, the calculation module 22 calculates, as a variation, thesecond difference between the standard deviation of brightness in thepast and the standard deviation of brightness at present for thecorresponding small region 100.

The calculation module 22 outputs the calculated brightness informationto the extraction module 23 and the detection module 25.

The extraction module 23 extracts, as the high-brightness region 200,the small region 100 in which the brightness average calculated by thecalculation module 22 is equal to or larger than a predetermined value.For example, the extraction module 23 extracts the high-brightnessregion 200 from among the small regions 100 included in thepredetermined region ROI, excluding a vehicle body region (body region)corresponding to the vehicle body. With this configuration, thehigh-brightness region 200 can be extracted with high accuracy throughgain control. The extraction module 23 outputs information on theextracted high-brightness region 200 to the decision module 24.

The decision module 24 decides on a brightness threshold value fordetecting the deposit region 500 corresponding to a light-blockingdeposit adhering to the lens of the camera 10, based on the area of thehigh-brightness region 200 extracted by the extraction module 23.

A decision process for a brightness threshold value by the decisionmodule 24 will now be described with reference to FIG. 4. FIG. 4 is adiagram illustrating an example of the threshold value information 31.In the graph illustrated in FIG. 4, the horizontal axis represents thebrightness average of the entire image and the vertical axis representsthe brightness threshold value.

The decision module 24 decides on a brightness threshold value to beused in the detection process by the detection module 25, based on thethreshold value information 31. Specifically, the decision module 24decides on a brightness threshold value based on the number ofhigh-brightness regions 200 extracted by the extraction module 23. Morespecifically, the decision module 24 decides on an upper limit value ofthe brightness threshold value in accordance with the number ofhigh-brightness regions 200. For example, the decision module 24 decideson the upper limit value such that the larger the number ofhigh-brightness regions 200 is, the higher the upper limit value is.

The decision module 24 then decides on a brightness threshold value in arange of the decided upper limit value, based on the brightness averageof the entire image. For example, when the brightness average of theentire image is “A”, the decision module 24 decides on a threshold valueTH1 as the brightness threshold value if the number of high-brightnessregions 200 is equal to or larger than a predetermined large-numberthreshold value (high-brightness regions: many).

When the brightness average of the entire image is “A”, the decisionmodule 24 decides on a threshold value TH2 as the brightness thresholdvalue if the number of high-brightness regions 200 is equal to or largerthan a predetermined intermediate threshold value (high-brightnessregions: intermediate).

When the brightness average of the entire image is “A”, the decisionmodule 24 decides on a threshold value TH3 as the brightness thresholdvalue if the number of high-brightness regions 200 is smaller than apredetermined small-number threshold value or zero (high-brightnessregions: few).

In this way, the decision module 24 decides on a brightness thresholdvalue based on the brightness average of the entire image (the entireregion of the captured image I) and the number (area) of high-brightnessregions 200, whereby a brightness threshold value can be decided on withhigh accuracy. Since the number of high-brightness regions 200 istreated as the area, an arithmetic process for calculating the area isunnecessary. That is, a brightness threshold value is decided on basedon the number of high-brightness regions 200, whereby the process loadon the control unit 2 can be suppressed.

Although in FIG. 4, the upper limit value of the brightness thresholdvalue is decided on by three levels (threshold values TH1, TH2, andTH3), the upper limit value may be decided on by two or less levels orby four or more levels.

The decision module 24 outputs the decided brightness threshold value tothe detection module 25.

The detection module 25 detects the deposit region 500 based on thebrightness threshold value decided on by the decision module 24.Specifically, the detection module 25 determines whether the smallregion 100 satisfies all of the following conditions (1) to (3). Whenthe small region 100 does not satisfy at least one of the conditions (1)to (3), the small region 100 is detected as a non-deposit region inwhich a deposit does not adhere.

(1) The first difference of the small region 100 is equal to or smallerthan a predetermined value.

(2) The second difference of the small region 100 is equal to or smallerthan a predetermined value.

(3) The brightness average of the small region 100 is equal to orsmaller than the brightness threshold value.

Subsequently, when the number of small regions 100 in which a countnumber indicating the number of times all of the conditions (1) to (3)above are satisfied is equal to or larger than a predetermined number isequal to or larger than a predetermined number, the detection module 25detects the predetermined number of small regions 100 as the depositregion 500. That is, the detection module 25 detects, as the depositregion 500, a set of the predetermined number of small regions 100 inwhich the state of satisfying all of the conditions (1) to (3) continuesa predetermined number of times or more.

The detection module 25 outputs information on the detected depositregion 500 to the flag output module 26.

The flag output module 26 outputs a deposit flag ON to the variousequipment 50 when the deposit region 500 is detected by the detectionmodule 25. On the other hand, the flag output module 26 outputs adeposit flag OFF to the various equipment 50 when the deposit region 500is not being detected by the detection module 25, that is, a non-depositregion is detected.

Referring now to FIG. 5 and FIG. 6, a procedure of a process performedby the deposit detection device 1 according to the embodiment will bedescribed. FIG. 5 is a flowchart illustrating a procedure of the wholeprocess performed by the deposit detection device 1 according to theembodiment. FIG. 6 is a flowchart illustrating a procedure of thedetection process performed by the deposit detection device 1 accordingto the embodiment.

First referring to FIG. 5, the whole process performed by the depositdetection device 1 according to the embodiment will be described.

As illustrated in FIG. 5, the preprocessing module 21 performspreprocessing on the captured image I (step S101). As used herein, thepreprocessing is a process of performing the gray-scale process and thethinning process and thereafter generating an integrated image based onpixel values of the reduced image.

Subsequently, the calculation module 22 divides a predetermined regionROI in the captured image I into small regions 100 (step S102).

Subsequently, the calculation module 22 calculates brightnessinformation indicating a feature amount of brightness for each of thesmall regions 100 (step S103). The feature amount of brightness is, forexample, an average value of brightness and a standard deviation ofbrightness.

Subsequently, the calculation module 22 calculates brightnessinformation indicating a feature amount of brightness of the entireimage (step S104).

Subsequently, the extraction module 23 extracts, as the high-brightnessregion 200, the small region 100 in which the brightness average of thesmall region 100 calculated by the calculation module 22 is equal to orlarger than a predetermined value (step S105).

Subsequently, the decision module 24 decides on a brightness thresholdvalue for the detection process by the detection module 25, based on thearea of the high-brightness region 200 extracted by the extractionmodule 23 (step S106).

Subsequently, the detection module 25 performs the detection process ofdetecting the deposit region 500, using the brightness threshold valuedecided on by the decision module 24 (step S107).

Subsequently, the flag output module 26 outputs a deposit flagcorresponding to the detection result by the detection module 25 to thevarious equipment 50 (step S108) and terminates the process.

Referring now to FIG. 6, the detection process performed by the depositdetection device 1 according to the embodiment will be described.

First, the detection module 25 determines whether the first differenceof the small region 100 is equal to or smaller than a predeterminedvalue (step S201).

If the first difference is equal to or smaller than a predeterminedvalue (Yes at step S201), the detection module 25 determines whether thesecond difference of the small region 100 is equal to or smaller than apredetermined value (step S202).

If the second difference is equal to or smaller than the predeterminedvalue (Yes at step S202), the detection module 25 determines whether thebrightness average of the small region 100 is equal to or smaller than abrightness value (step S203).

If the brightness average of the small region 100 is equal to or smallerthan the brightness value (Yes at step S203), the detection module 25increments a count number indicating the number of times the conditionsof step S201 to step S203 are satisfied, and determines whether theincremented count number is equal to or larger than a predeterminednumber (step S204).

If the count number is equal to or larger than the predetermined number(Yes at step S204), the detection module 25 detects the small region 100as a region (small region 100) that satisfies the conditions (stepS205).

Subsequently, the detection module 25 determines whether the number ofregions that satisfy the conditions is equal to or larger than apredetermined number (step S206).

If the number of regions that satisfy the conditions is equal to orlarger than the predetermined number (Yes at step S206), the detectionmodule 25 detects the predetermined number of small regions 100 as thedeposit region 500 (step S207).

Subsequently, if the deposit region 500 is detected, the flag outputmodule 26 outputs the deposit flag ON (step S208) and terminates theprocess.

On the other hand, if the first difference exceeds the predeterminedvalue at step S201 (No at step S201), the detection module 25 detectsthe small region 100 as a region that does not satisfy the conditions(step S209).

If the second difference exceeds the predetermined value at step S202(No at step S202), the detection module 25 proceeds to step S209.

If the brightness average exceeds the brightness threshold value at stepS203 (No at step S203), the detection module 25 proceeds to step S209.

If the count number is smaller than the predetermined number at stepS204 (No at step S204), the detection module 25 proceeds to step S209.

If the number of regions that satisfy the conditions is smaller than thepredetermined number at step S206 (No at step S206), the detectionmodule 25 detects the predetermined region ROI as a non-deposit region(step S210).

Subsequently, if a non-deposit region is detected, the flag outputmodule 26 outputs a deposit flag OFF (step S211) and terminates theprocess.

As described above, the deposit detection device 1 according to theembodiment includes the calculation module 22, the extraction module 23,and the decision module 24. The calculation module 22 calculates abrightness average for each of small regions 100 into which apredetermined region ROI of the captured image I captured by the imagingdevice (camera 10) is divided. The extraction module 23 extracts, as thehigh-brightness region 200, the small region 100 in which the brightnessaverage calculated by the calculation module 22 is equal to or largerthan a predetermined value. The decision module 24 decides on abrightness threshold value for detecting the deposit region 500corresponding to a light-blocking deposit adhering to the imagingdevice, based on the area of the high-brightness region 200 extracted bythe extraction module 23. With this configuration, a deposit can bedetected with high accuracy.

In the foregoing embodiment, the captured image I captured by a cameramounted on a vehicle is used. However, the captured image I may be, forexample, a captured image I captured by a security camera or a camerainstalled on a street lamp. That is, the captured image I may be anycaptured image captured by a camera with a lens to which a deposit mayadhere.

Although the invention has been described with respect to specificembodiments for a complete and clear disclosure, the appended claims arenot to be thus limited but are to be construed as embodying allmodifications and alternative constructions that may occur to oneskilled in the art that fairly fall within the basic teaching herein setforth.

What is claimed is:
 1. A deposit detection device comprising: acalculation module configured to calculate a brightness average for eachof small regions into which a predetermined region of an image capturedby an imaging device is divided; an extraction module configured toextract, as a high-brightness region, the small region in which thebrightness average calculated by the calculation module is equal to orlarger than a predetermined value; and a decision module configured todecide on a brightness threshold value for detecting a deposit regioncorresponding to a deposit adhering to the imaging device, based on anarea of the high-brightness region extracted by the extraction module.2. The deposit detection device according to claim 1, wherein thedecision module decides on the brightness threshold value based onnumber of the high-brightness regions extracted by the extractionmodule.
 3. The deposit detection device according to claim 1, whereinthe calculation module calculates a brightness average of the entirepredetermined region, and the decision module decides on the brightnessthreshold value, based on a brightness average of an entire region ofthe image and the area of the high-brightness region.
 4. The depositdetection device according to claim 1, wherein the extraction moduleextracts the high-brightness region from among the small regionsexcluding a body region corresponding to a vehicle body, in thepredetermined region.
 5. A deposit detection method comprising:calculating a brightness average for each of small regions into which apredetermined region of an image captured by an imaging device isdivided; extracting, as a high-brightness region, the small region inwhich the brightness average calculated at the calculating is equal toor larger than a predetermined value; and deciding on a brightnessthreshold value for detecting a deposit region corresponding to adeposit adhering to the imaging device, based on an area of thehigh-brightness region extracted at the extracting.